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The  Honorable  Charles  Bolden 
Administrator 

National  Aeronautics  and  Space  Administration 
300  E  Street,  SW 
Washington,  DC  20024-3210 


Subject:  Additional  Cost  Transparency  and  Design  Criteria  Needed  for  National  Aeronautics 

and  Space  Administration  (NASA)  Projects 

Dear  Administrator  Bolden: 

Today,  GAO  published  its  third  annual  assessment  of  selected  large-scale  NASA  projects/ 
During  this  assessment  we  identified  several  issues  that  merit  your  management  attention. 

The  federal  government  faces  real  fiscal  limitations  and  will  have  to  make  difficult  choices 
about  upcoming  priorities.  This  reality  makes  it  more  important  than  ever  that  NASA  manage 
its  programs  and  projects  as  efficiently  and  effectively  as  possible  and  within  a  budget  that 
over  recent  years  has  remained  relatively  constant.  It  will  also  require  that  NASA  make 
tough  decisions  about  which  projects  to  fund  among  core  missions  in  science,  aeronautics, 
and  human  space  flight  and  exploration.  Our  work  over  the  past  three  years  has  shown  that 
NASA’s  major  projects  are  frequently  approved  without  evidence  of  a  sound  business  case — 
ensuring  a  match  between  requirements  and  resources — and,  therefore,  cost  more  and  take 
longer  to  develop  than  planned.  Our  March  2011  assessment  found  that  13  NASA  projects 
that  established  baselines  prior  to  fiscal  year  2009  had  experienced  an  average  cost  growth  of 
almost  55  percent,  with  a  combined  increase  in  development  costs  of  almost  $2.5  billion 
from  their  baselines  established  at  their  Confirmation  Review.  While  NASA  has  taken  steps 
over  recent  years  to  help  improve  its  acquisition  management  through  several  initiatives 
aimed  at  improving  cost  estimating  and  management  oversight,  the  overall  outcomes  of  these 
efforts  will  take  time  to  become  apparent.  Based  on  the  findings  of  our  past  three 
assessments,  we  are  recommending  that  NASA  (1)  provide  increased  transparency  into 
project  costs  to  the  Congress  to  conduct  oversight  and  ensure  earlier  accountability  and  (2) 
develop  a  common  set  of  measurable  and  proven  criteria  to  assess  the  design  stability  of 
projects  before  proceeding  into  later  phases  of  development. 


'  GAO,  NASA:  Assessments  of  Selected  Large-Scale  Projects,  GAO-1 1-239SP  (Washington,  D.C.:  Mar.  3,  2011). 

^  The  confirmation  review  is  the  point  at  which  cost  and  schedule  baselines  are  established  and  approved  for  the  project. 
Project  progress  is  measured  against  these  baselines. 
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Lack  of  Transparency  Into  Early  Project  Development  Costs 


As  we  have  reported,  NASA  does  not  provide  enough  transparency  into  costs  during  early 
projeet  development  to  provide  the  Congress  with  suffieient  information  to  conduet  oversight 
and  ensure  earlier  accountability.^  While  there  is  a  need  to  allow  projeets  a  period  of  time  for 
discovery  and  to  pursue  different  eoneepts,  projects  can  spend  signifieant  sums  of  time  and 
money  before  they  enter  implementation"^  and  are  held  accountable  to  a  formal  baseline.  For 
example,  authorized  in  2005  and  2006  respeetively,  the  Ares  and  Orion  projeets^  have  spent 
over  $9  billion  combined,  but  neither  has  reached  implementation.  Furthermore,  the  James 
Webb  Spaee  Telescope  was  authorized  in  1999,  reached  implementation  in  2008,  and  spent 
nearly  $2  billion  during  those  9  years.  The  Senate  Committee  on  Commerce,  Seience,  and 
Transportation  recently  asked  GAO  to  testify  on  how  NASA  ean  improve  its  overall 
transpareney  and  aceountability. 

Currently,  NASA  does  not  provide  to  the  Congress  cost  and  schedule  information  for 
projeets  in  the  early,  eritieal  phases  of  development  and  makes  this  information  public  only 
after  the  projeets  have  been  formally  approved  to  enter  implementation.  Projects  establish 
preliminary  eost  estimates  in  the  formulation  phase;  these  estimates,  however,  are  for 
planning  purposes  only  as  they  enable  NASA  decision-makers  to  better  manage  the  overall 
portfolio  of  projeets.  NASA  does  not  report  deviations  from  these  early  estimates  to  the 
Congress.  Although  progress  is  not  measured  or  reported  externally  against  early  planning 
baselines,  cost  growth  and  schedule  delays  can  and  do  occur  during  the  formulation  phase. 
NASA’s  internal  analysis  of  past  projeets  indieates  that  there  is  an  average  of  14  pereent 
growth  in  the  development  cost  estimates  during  the  formulation  phase.  Additionally,  NASA 
does  not  report  information  on  the  funding  that  has  been  authorized  to  date  for  the  projeets  in 
formulation,  as  it  does  in  its  annual  budget  submission  for  projects  in  implementation. 
Additional  insight  into  costs  would  provide  eritieal  information  to  help  mateh  projeet 
requirements  to  resources  and  could  better  enable  the  Congress  to  make  more  informed 
deeisions  when  approving  the  projeets  through  the  annual  appropriations  proeess.  Our  prior 
reports  have  shown  that  early  insight  into  projeet  progress  ean  put  deeision-makers  in  a  better 
position  to  ensure  projeets  have  the  resourees  necessary  for  suecess. 


Laek  of  Design  Metric  May  Contribute  to  Projeet  Cost  Growth 


During  the  eourse  of  our  past  three  reviews  of  large-seale  NASA  projects  and  other  work 
examining  NASA’s  aequisition  management,^  we  found  that  NASA  does  not  use  a  eommon. 


^  GAO,  NASA:  Issues  Implementing  the  NASA  Authorization  Act  of  2010,  GAO-1 1-216T  (Washington,  D.C.:  Dec.  1,  2010); 
GAO-11-239SP. 

There  are  two  phases  in  NASA’s  project  life  cycle — the  formulation  phase  and  the  implementation  phase.  In  the 
formulation  phase,  the  project  defines  requirements — what  the  project  is  being  designed  to  do — matures  technology, 
establishes  a  schedule,  estimates  costs,  and  produces  a  plan  for  implementation.  In  the  implementation  phase,  the  project 
carries  out  these  plans,  performing  final  design  and  fabrication  as  well  as  testing  components  and  system  assembly, 
integrating  these  components  and  testing  how  they  work  together,  and  launching  the  project.  This  phase  also  includes  the 
period  from  project  launch  through  mission  completion. 

^  As  part  of  the  Constellation  Program,  the  Ares  I  Crew  Launch  Vehicle  was  designed  to  carry  the  Orion  Crew  Exploration 
Vehicle  into  low-Earth  orbit  for  missions  to  the  International  Space  Station  and  the  Moon. 

*  GAO,  NASA:  Implementing  a  Knowledge-Based  Acquisition  Framework  Could  Lead  to  Better  Investment  Decisions  and 
Project  Outcomes,  GAO-06-218  (Washington,  D.C.:  Dec.  21,  2005);  GAO,  NASA:  Assessments  of  Selected  Large-Scale 
Projects,  GAO-09-306SP  (Washington,  D.C.:  Mar.  2,  2009);  GAO,  NASA:  Assessments  of  Selected  Large-Scale  Projects, 
GAO-10-227SP  (Washington,  D.C.:  Feb.  1,  2010);  GAO-1 1-239SP;  and  GAO- 11 -2 16T. 
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proven  metric  to  assess  design  stability  before  allowing  programs  to  move  from  the  design 
phase  to  the  test  and  integration  phase^  of  the  development  process.  NASA’s  acquisition 
policy  does  not  specify  a  metric  by  which  a  project’s  design  stability  is  measured  at  the 
critical  design  review.*  Guidance  in  NASA’s  Systems  Engineering  Handbook,  however, 
mirrors  an  accepted  best  practice^  that  at  least  90  percent  of  engineering  drawings  should  be 
releasable  by  the  critical  design  review,  although  projects  are  not  required  to  follow  this 
guidance. We  have  found  that  if  design  stability  is  not  achieved  at  the  critical  design 
review,  but  product  development  continues,  costly  re-designs  to  address  changes  to  project 
requirements  and  unforeseen  challenges  can  occur.  As  shown  in  figure  1,  nearly  all  of  the 
projects  we  reviewed  over  the  last  three  years  held  their  critical  design  review  without  90 
percent  of  engineering  drawings  being  releasable — failing  to  meet  NASA’s  Systems 
Engineering  Handbook  guidance  and  our  best  practices  criteria  for  design  stability.  The  only 
two  projects  in  our  2011  review  that  met  the  design  criteria — Orbiting  Carbon  Observatory-2 
and  Tracking  and  Data  Relay  Satellite  Replenishment — to  date  have  experienced  no  cost 
growth;  however,  these  projects  are  based  heavily  on  previous  designs.  Ten  projects  did  not 
meet  the  design  criteria  and,  of  those,  nine  had  experienced  development  cost  growth  since 
their  baseline. 


’  During  the  design  phase,  the  project  completes  its  preliminary  design  and  technology  development  and  begins  fabrication 
of  test  and  flight  article  components,  assemblies,  and  subsystems.  In  the  test  and  integration  phase,  the  project  initiates 
system  assembly,  integration  and  test  in  preparation  for  flight. 

^According  to  NASA  Interim  Directive  for  NASA  Procedural  Requirements  7120.5D,  Table  2-7  (Sept.  22,  2009),  the 
critical  design  review  demonstrates  that  the  maturity  of  the  design  is  appropriate  to  support  proceeding  with  full-scale 
fabrication,  assembly,  integration,  and  test,  and  that  the  technical  effort  is  on  track  to  complete  the  flight  and  ground  system 
development  and  mission  operations  in  order  to  meet  mission  performance  requirements  within  the  identified  cost  and 
schedule  constraints.  Progress  against  management  plans,  budget,  and  schedule,  as  well  as  risk  assessments,  are  presented. 

^  GAO,  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early  Improves  Acquisition  Outcomes,  GAO-02- 
701  (Washington,  D.C.:  July  15,  2002). 

Engineering  drawings  are  considered  to  be  a  good  measure  of  the  demonstrated  stability  of  a  product’s  design  because  the 
drawings  represent  the  language  used  by  engineers  to  communicate  to  the  manufacturers  the  details  of  a  new  product 
design — ^what  it  looks  like,  how  its  components  interface,  how  it  functions,  how  to  build  it,  and  what  critical  materials  and 
processes  are  required  to  fabricate  and  test  it.  Once  the  design  of  a  product  is  finalized,  the  drawing  is  "releasable". 
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Figure  1:  2009-2011  Trend  of  Design  Criteria  for  Projects  in  Implementation 


Percent 


Year 


Projects  in  implementation  meeting  design  criteria 


Projects  in  implementation  not  meeting  design  criteria 


Source:  GAO  analysis  data  provided  by  NASA. 


Note:  In  our  2009,  2010,  and  201 1  assessments  we  reviewed  10,9,  and  1 2  projects,  respectively,  that  were  in  implementation 
and  had  held  their  critical  design  review. 


Some  of  the  projects  we  reviewed  in  the  past  three  years  identified  other  activities  that 
occurred  prior  to  the  critical  design  review  as  evidence  of  design  stability.  In  addition  to 
releasable  engineering  drawings,  NASA  relies  on  subject  matter  experts  in  the  design  review 
process  and  other  methods  to  assess  design  stability.  For  example,  a  panel  of  experts  within 
NASA  provides  an  assessment  of  the  technical  and  programmatic  approach,  risk  posture,  and 
progress  against  the  project  baseline  at  key  decision  points  to  be  assured  that  the  project  has  a 
stable  design.  Some  projects  rely  on  using  engineering  models  and  engineering  test  units  to 
assess  design  stability.  According  to  NASA’s  systems  engineering  policy,  at  the  critical 
design  review  the  project’s  design  should  be  stable  enough  to  support  full  scale-fabrication, 
assembly,  integration,  and  test.^^  However,  a  recent  study  by  the  National  Research 
Council  found  that  the  critical  design  review  milestone  for  many  NASA  missions  may  be 
held  prematurely — driven  by  schedule  rather  than  driven  by  design  maturity.  Furthermore, 
we  found  that  the  majority  of  the  12  projects  in  our  March  2011  review  that  had  held  their 
critical  design  review  experienced  increases  in  the  number  of  engineering  drawings  that  were 
expected  after  that  review — an  indication  of  an  unstable  design  after  the  critical  design 
review.  This  is  particularly  evident  for  the  four  projects  that  held  their  critical  design  reviews 
prior  to  fiscal  year  2009  — the  projects  that  entered  the  test  and  integration  phase  in  2009  or 


"NASA  Procedural  Requirement  7 123.1  A,  NASA  Systems  Engineering  Processes  and  Requirements  Appendix  G, 
paragraph  G.8  (Mar.  26,  2007). 

Controlling  Cost  Growth  of  NASA  Earth  and  Space  Science  Missions,  National  Research  Council,  The  National 
Academies,  Washington  D.C.  2010. 

"  Sixteen  projects  in  our  2011  review  were  in  the  implementation  phase.  Of  those,  twelve  projects  had  held  their  critical 
design  reviews  and  could  provide  information  on  the  number  of  design  drawings. 
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before  and  have  more  of  a  history  to  track  variances.  As  shown  in  figure  2  below,  the  four 
projects,  on  average,  had  a  107  percent  increase  in  expected  engineering  drawings  after  the 
critical  design  review  after  having  only  36  percent  of  drawings  releasable  at  that  review.  The 
remaining  eight  projects  that  held  their  critical  design  review  in  fiscal  year  2009  or  later  have 
not  reported  a  large  increase  in  expected  drawings.  All  of  these  projects  will  enter  or  have 
entered  the  test  and  integration  phase  in  2010  or  later.  A  National  Research  Council  study 
states  that  approval  of  an  immature  design  at  the  critical  design  review  can  cause  problems 
during  test  and  integration  such  as  integration  difficulties  and  late  design  changes.  An 
increase  in  design  drawings  is  one  indication  of  late  design  changes. 


Figure  2:  Comparison  of  Design  Drawing  Increase  for  Projects  with  Critical  Design 
Review  (CDR)  prior  to  and  since  Fiscal  Year  2009 
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Source:  GAO  analysis  of  data  provided  by  NASA. 


Regardless  of  how  stability  is  measured,  consistent,  proven,  and  quantitative  measures 
employed  at  the  critical  design  review — such  as  the  percentage  of  engineering  drawings — 
can  provide  evidence  that  the  design  is  stable  and  will  meet  performance  requirements.  These 
measures  can  also  be  an  indication  to  decision-makers  that  the  requisite  knowledge  has  been 
attained  to  allow  the  project  to  proceed  in  its  development  lifecycle  and  better  enable  them  to 
assess  the  performance  of  individual  projects  against  the  overall  portfolio  of  projects. 


We  began  tracking  the  percentage  of  drawing  releasable  as  a  measure  of  the  design  stability  of  selected  large-scale  NASA 
projects  in  our  2009  assessment.  Since  then,  NASA  provides  drawing  information  on  projects  included  in  our  yearly 
assessment  on  a  semiannual  basis. 
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Conclusions 


Over  the  past  several  years,  our  assessments  have  shown  that  NASA’s  major  aequisitions 
have  been  marked  by  cost  overruns  and  sehedule  delays.  Given  the  multiple  priorities  that 
NASA  must  balanee — eoupled  with  the  eonstraints  that  will  eontinue  to  bind  the 
government’s  fiseal  resourees  for  many  years — ensuring  that  NASA  is  basing  its  aequisition 
deeisions  on  eorrect,  timely,  and  consistent  information  is  oritieal.  While  NASA  has 
undertaken  several  initiatives  aimed  at  improving  its  aequisition  management,  the  laek  of 
transpareney  into  the  eost  of  projeets  in  the  early,  eritieal  phases  of  development  leaves  the 
Congress  with  ineomplete  knowledge  to  inform  its  oversight  and  ensure  earlier 
aeeountability.  While  there  is  a  need  to  allow  projeets  a  period  of  time  for  diseovery  and  to 
pursue  different  concepts,  particularly  highly  complex  efforts,  inadequate  transpareney  into 
projeet  progress  for  what  sometimes  amounts  to  5  or  more  years  ean  preelude  effeetive 
oversight  and  aeeountability  and  make  it  even  more  difficult  to  stop  projects  that  are  not  on 
track  to  meet  the  ageney’s  goals  with  available  resources.  Finally,  without  a  standard  design 
metrie  to  measure  the  projeets’  progress  at  crucial  points  in  the  development  life  cycle, 
NASA  eannot  be  assured  that  its  deeisions  will  result  in  the  best  possible  return  on  its 
investments. 

Reeommendations  for  Exeeutive  Action 


To  provide  inereased  transpareney  into  projeet  risks,  we  reeommend  that  you 

•  Direet  the  Office  of  the  Chief  Finaneial  Offieer  (OCFO)  to  provide  more  transpareney 
into  projeet  eosts  in  the  early,  eritieal  phases  of  development  so  that  the  Congress  has 
suffieient  information  to  eonduet  oversight  and  ensure  earlier  aeeountability.  Speoifieally, 
the  OCFO  should  provide  progress  reports  for  NASA  spaee  flight  programs  and  projects 
in  formulation  that  include  information  on  eumulative  prior  budget  authority  and  eurrent 
eost  ranges  in  NASA’s  annual  budget  submission  to  the  Congress  to  enhanee  the 
knowledge  with  whieh  they  base  funding  deeisions  for  NASA. 

To  provide  NASA  deeision-makers  with  a  meehanism  to  eonsistently  and  aecurately  judge 
design  stability,  we  recommend  that  you 

•  Direet  the  Offiee  of  the  Chief  Engineer  to  develop  a  eommon  set  of  measurable  and 
proven  eriteria,  such  as  the  pereentage  of  releasable  design  drawings,  to  assess  design 
stability  and  to  allow  decision-makers  to  make  more  informed,  eonsistent  determinations 
of  approval  for  projeets  to  proeeed  from  the  final  design  phase  to  the  assembly, 
integration,  and  test  phase  of  the  development  proeess  and  amend  NASA’s  systems 
engineering  poliey,  aeeordingly. 

Ageney  Comments  and  Our  Evaluation 

In  written  eomments  on  a  draft  of  this  report,  the  Deputy  Administrator  of  NASA  partially 
coneurred  with  our  first  recommendation  and  eoneurred  with  our  seeond  the 
reeommendation.  NASA’s  eomments  are  reprinted  in  the  Enelosure. 
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NASA  partially  concurred  with  our  recommendation  to  provide  more  transpareney  into 
projeet  eosts  in  the  early,  eritieal  phases  of  development  in  order  to  provide  the  Congress 
suffieient  information  to  eonduet  oversight  and  ensure  earlier  aeeountability.  While  NASA 
agreed  that  elarity  is  needed  into  life-eycle  eost  range  estimates,  it  stated  that  the  estimates 
developed  during  formulation  are  not  a  baseline,  and  therefore  not  a  basis  for  measurement 
of  eost  growth.  Preliminary  eost  estimates  are  used  by  NASA  to  determine  whether  there  is  a 
good  mateh  between  requirements  and  resourees  when  designing  a  projeet.  NASA  stated  that 
it  eurrently  provides  these  preliminary  eost  range  estimates  upon  request,  and  is  eonsidering 
providing  inereased  information  for  projeets  in  early  formulation  in  the  annual  budget 
submission  to  the  Congress.  NASA  did  not  eommit,  however,  to  providing  information  to  the 
Congress  on  the  eumulative  prior  budget  authority  for  projects  in  formulation,  as  we 
recommended.  We  understand  that  preliminary  eost  estimates  are  distinct  from  the  baseline 
eosts  of  the  projeet  and  are  not  intended  as  a  basis  to  measure  eost  growth.  These  early  eost 
estimates  do,  however,  provide  the  basis  of  the  finaneial  investment  that  the  ageney  is 
eommitting  the  government  to  and  should  be  transparent  to  Congressional  decision-makers 
for  their  deliberation  and  deeision-making.  The  laek  of  transpareney  into  project  progress 
early  in  development  ean  preelude  the  ability  of  the  Congress  to  eonduet  oversight  and 
ensure  aeeountability  and  make  it  even  more  diffieult  to  stop  projeets  that  are  not  on  traek  to 
meet  the  ageney’ s  goals  with  available  resourees.  Similarly,  given  NASA’s  various 
aeeounting  ehanges  throughout  the  past  deeade,  it  is  difficult  if  not  impossible  for  the 
Congress  and  parties  external  to  NASA  to  determine  the  eumulative  budget  authority  or  the 
investment  to  date  that  has  been  given  to  a  particular  project.  Providing  this  type  of  budget 
information  would  allow  Congressional  deeision-makers  insight  into  the  eommitments  to  the 
projeet  thus  far  to  make  informed  deeisions  about  their  eontinued  approval. 


NASA  eoneurred  with  our  seeond  reeommendation  that  the  ageney  develop  a  eommon  set  of 
measurable  and  proven  eriteria  to  assess  design  stability.  NASA  stated  that  the  drawing 
release  metrie  is  a  legaey  standard  developed  prior  to  the  use  of  computerized  drawings,  and 
henee  does  not  take  into  aeeount  improvements  due  to  the  use  of  this  teehnology.  We 
aeknowledge  this  point,  but  our  analysis  of  NASA  projeets  shows  that  those  projeets  that 
have  met  or  eome  elose  to  meeting  the  best  practices  drawing  release  metrie  have  fared  better 
with  regard  to  eost  and  schedule  than  those  projeets  that  did  not  come  elose  to  meeting  the 
metrie.  NASA  reeognizes  the  need  for  metries  for  greater  insight  into  projeet  formulation  and 
development  and  stated  that  it  is  developing  a  eommon  set  of  measurable  and  proven  eriteria 
to  assess  design  maturity,  whieh  it  will  provide  to  GAO  no  later  than  March  31,  201 1. 

Scope  and  Methodology 

The  majority  of  our  findings  in  this  eorrespondenee  were  based  on  work  eompleted  as  part  of 
our  2011  Assessments  of  Selected  Large-Scale  Projects.  In  the  assessment,  we  reviewed  eost 
growth  and  design  maturity  of  21  projeets,  eaeh  with  an  estimated  life-eyele  eost  of  over 
$250  million.  We  eondueted  interviews  with  project  officials  and  examined  the  eurrent  phase 
of  a  projeet’s  development  and  how  eaeh  projeet  was  advaneing.  NASA  provided  updated 
eost  and  sehedule  data  as  of  February  2011  for  16  of  the  21  projeets.  We  reviewed  and 
eompared  that  data  to  previously  established  cost  and  schedule  baselines.  We  assessed  eaeh 
projeet’s  eost  and  schedule  and  eharaeterized  growth  in  either  as  signifieant  if  it  exceeded  the 
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thresholds  that  trigger  reporting  to  the  Congress  under  the  law.  We  also  reviewed  NASA's 
February  2011  budget  submission  to  the  Congress  and  noted  differenees  between  eost 
information  provided  for  projeets  in  implementation  versus  eost  information  provided  for 
projeets  in  formulation.  In  addition,  NASA  provided  eost  and  sehedule  information  from 
previously  reported  projeets  that  we  used  for  historieal  analysis.  Through  a  standardized  data 
eolleetion  instrument,  we  gathered  basie  information  about  the  projeets  as  well  as  eurrent  and 
projeeted  development  aetivities  for  those  projeets,  ineluding  information  eoneeming  the 
projeet’s  engineering  design  drawings.  We  used  this  data  to  assess  design  stability  against 
GAO’s  established  eriteria  for  knowledge-based  aequisitions  and  on  other  GAO  work  on 
system  aequisitions.  We  reviewed  and  diseussed  NASA  polieies  and  in-house  praetiees 
related  to  evaluating  the  design  stability  of  eaeh  projeet.  We  took  appropriate  steps  to  address 
data  reliability. 

We  eondueted  this  performanee  audit  in  February  2011  in  aeeordanee  with  generally 
aeeepted  government  auditing  standards.  Those  standards  require  that  we  plan  and  perform 
the  audit  to  obtain  suffieient,  appropriate  evidenee  to  provide  a  reasonable  basis  for  our 
findings  and  eonelusions  based  on  our  audit  objeetives.  We  believe  that  the  evidenee 
obtained  provides  a  reasonable  basis  for  our  findings  and  eonelusions  based  on  our  audit 
objeetives. 

We  are  sending  eopies  of  this  report  to  interested  eongressional  eommittees.  In  addition,  the 
report  will  be  available  at  no  eharge  on  GAO’s  Web  site  at  http://www.gao.gov.  If  you  have 
any  questions,  please  eontaet  me  at  (202)  512-4841  or  ohaplaino@gao.gov.  Contaot  points 
for  our  Offioes  of  Congressional  Relations  and  Publio  Affairs  may  be  found  on  the  last  page 
of  this  letter. 

Key  oontributors  to  this  report  were  Shelby  S.  Oakley,  Assistant  Direotor;  Jessioa  M. 
Berkholtz;  Riohard  A.  Cederholm;  Laura  Greifner,  Kristine  R.  Massinger,  and  Roxanna  T. 
Sun. 


Cristina  Chaplain 
Direotor 

Aoquisition  and  Souroing  Management 


(120961) 


NASA  is  required  to  report  to  Congress  if  development  cost  of  a  program  is  likely  to  exceed  the  baseline  estimate  by  15 
percent  or  more,  or  if  a  milestone  is  likely  to  be  delayed  by  6  months  or  more.  42  U.S.C.  §  16613(d). 
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Enclosure:  Comments  from  the  National  Aeronautics  and  Space  Administration 


National  Aeronautics  and 
Space  Administration 

Office  of  Ihe  Administrator 

Washington,  DC  20546-0001 

February  23,  2011 


Ms.  Christina  Chaplain 
Director 

Acquisition  and  Sourcing  Management 
United  States  Government  Accountability  Office 
Washington,  DC  20548 

Dear  Ms.  Chaplain: 

The  National  Aeronautics  and  Space  Administration  (NASA)  appreciates  the 
opportunity  to  review  the  Government  Accountability  Office  (GAO)  draft 
correspondence  entitled  “Additional  Cost  Transparency  and  Design  Criteria  Needed  for 
National  Aeronautics  and  Space  Administration  (NASA)  Projects”  (GAO-1 1-364R). 
NASA  values  the  continued  open  communications  between  NASA  and  the  GAO  team  on 
the  annual  assessment  of  large-scale  projects  and  appreciates  the  constructive  comments 
arising  as  a  result  of  this  effort. 

NASA  agrees  with  the  GAO’s  concern  about  managing  programs  and  projects  as 
efficiently  and  effectively  as  possible,  especially  within  a  budget  that  is  likely  to  be 
constrained  due  to  the  fiscal  limitations  currently  faced  by  all  Federal  Government 
agencies.  Consequently,  NASA  remains  dedicated  to  continuous  improvement  of 
acquisition  management  processes  and  performance.  We  are  pleased  that  GAO 
recognizes  NASA’s  ongoing  efforts  to  mitigate  acquisition  management  risk  and  lay  a 
stronger  foundation  for  reducing  project  cost  and  schedule  growth.  As  was  highlighted  in 
the  annual  assessment,  NASA  implemented  a  Joint  Cost  and  Schedule  Confidence  Level 
(JCL)  policy  in  2009  to  increase  the  likelihood  of  project  success  at  the  specified  funding 
level.  Since  institution  of  this  poUcy,  the  average  cost  growth  for  NASA’s  large  scale 
projects  has  remained  below  the  1 5  percent  threshold  which,  if  breached,  requires  special 
notification  to  Congress.  NASA  will  continue  to  assess  the  effect  of  utilizing  JCLs  to 
manage  project  cost  and  schedule  growth.  With  the  completion  and  launch  of  three 
missions  in  20 1 1 ,  NASA  will  have  a  better  measure  of  the  impact  of  our  acquisition 
management  improvement  efforts  over  the  last  five  years. 

In  the  draft  report,  GAO  makes  two  recommendations  to  the  Administrator 
intended  to  increase  transparency  into  project  risks  and  to  provide  NASA  decision 
makers  with  a  mechanism  to  consistently  and  accurately  judge  design  stability, 
specifically; 

Recommendation  1:  Direct  the  Office  of  the  Chief  Financial  Officer  (OCFO)  to  provide 
more  transparency  into  project  costs  in  the  early,  critical  phases  of  development  so  that 
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the  Congress  has  sufficient  information  to  conduct  oversight  and  ensure  earlier 
accountability.  Specifically,  the  OCFO  should  provide  progress  reports  for  NASA  space 
flight  programs  and  projects  in  formulation  that  include  information  on  cumulative  prior 
budget  authority  and  current  cost  ranges  in  NASA’s  annual  budget  submission  to  the 
Congress  to  enhance  the  knowledge  with  which  they  base  funding  decisions  for  NASA. 

Management’s  Response:  Partially  concur.  NASA  agrees  with  the  need  to  provide 
clarity  on  life-cycle  cost  range  estimates,  but  the  estimates  developed  during  formulation 
are  not  a  baseline,  and  therefore  not  a  basis  for  measurement  of  cost  growth.  As 
recognized  by  GAO,  these  estimates  are  used  by  NASA  decision  makers  to  manage  the 
overall  portfolio  of  projects.  Furthermore,  these  preliminary  estimates  are  used  to  help 
NASA  determine  whether  there  is  a  good  match  between  requirements  and  resources 
given  the  project  design  trades  at  that  point  in  time.  The  decision  to  approve  a  mission  to 
proceed  to  development  involves  a  complex  balance  between  a  mission’s  technical 
design,  scientific  and  technical  priorities,  schedule,  available  resources  within  a  portfolio 
of  projects,  and  overall  risk  assessment  of  mission  success  given  the  requirements  and 
constraints.  Trades  between  all  of  these  areas  may  or  may  not  lead  to  growth  in  the  cost 
estimate.  When  a  project  is  approved  to  proceed  to  development,  these  trades  are 
complete;  NASA  confirms  that  the  requirements  match  the  available  resources,  and 
commits  to  complete  the  project  within  the  life-cycle  cost  estimate  based  on  this  match. 

NASA  currently  provides  preliminary  cost  range  estimates,  inclusive  of  prior  year 
costs,  for  projects  that  have  passed  Key  Decision  Point  B  (KDP-B),  upon  request.  For 
example,  these  ranges  are  published  annually  as  part  of  the  GAO’s  assessment  of 
NASA’s  large-scale  projects.  NASA  is  also  considering  providing  increased  information 
for  projects  in  early  formulation  in  the  aimual  budget  submission. 

Recommendation  2:  Direct  the  Office  of  the  Chief  Engineer  to  develop  a  common  set 
of  measurable  and  proven  criteria,  such  as  the  percentage  of  releasable  design  drawings, 
to  assess  design  stability  and  to  allow  decision-makers  to  make  more  informed,  consistent 
determinations  of  approval  for  projects  to  proceed  from  the  final  design  phase  to  the 
assembly,  integration,  and  test  phase  of  the  development  process  and  amend  NASA’s 
systems  engineering  policy,  accordingly. 

Management’s  Response:  Concur.  NASA  agrees  that  projects  should  have  a  stable 
design  before  starting  assembly,  integration,  and  test.  Standing  Review  Boards  (SRB) 
comprised  of  independent  technical  and  programmatic  experts  are  asked  to  assess  design 
stability  at  Preliminary  Design  Review  (PDR)  and  Critical  Design  Review  (CDR)  using  a 
number  of  criteria  that  are  outlined  in  NASA’s  system  engineering  and  independent 
review  policies.  The  drawing  release  metric  referenced  in  NASA’s  Systems  Engineering 
Handbook  is  a  legacy  standard  developed  prior  to  the  use  of  computerized  drawings  and 
hence  does  not  take  into  account  improvements  due  to  the  use  of  this  technology.  For 
this  reason,  although  NASA  continues  to  use  drawing  release  as  one  of  many  indicators, 
the  Agency  does  not  place  as  much  emphasis  on  drawing  release  when  considered 
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holistically  along  with  other  criteria  to  assess  design  stability.  NASA  does  recognize  the 
need  for  metrics  for  greater  insight  into  project  formulation  and  development.  NASA 
concurs  with  GAO’s  recommendation  to  develop  a  common  set  of  criteria  to  assess 
design  stability.  NASA  is  developing  a  common  set  of  measurable  and  proven  criteria  to 
assess  design  maturity.  The  metrics  will  be  provided  to  GAO  no  later  than  March  31, 
2011. 

NASA  will  continue  to  follow  through  with  oiur  new  policies  and  management 
attention  on  cost  and  schedule  growth  and  project  management  in  the  coming  year.  We 
are  committed  to  continuous  improvement  in  order  to  explore  and  utilize  space  in  an 
affordable  way  for  the  benefit  of  the  Nation.  To  this  end,  we  look  forward  to  continuing 
to  work  with  the  GAO  to  measure  and  improve  our  performance  and  management 
practices. 

Thank  you  for  the  opportunity  to  comment  on  this  draft  report.  If  you  have  any 
questions  or  require  additional  information,  please  contact  Katie  Gallagher  at 
(202)358-2185. 


Sincerely, 


Lori  B.  Garver 
Deputy  Administrator 


(120961) 
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This  is  a  work  of  the  U.S.  government  and  is  not  subject  to  copyright  protection  in  the 
United  States.  The  pubiished  product  may  be  reproduced  and  distributed  in  its  entirety 
without  further  permission  from  GAO.  However,  because  this  work  may  contain 
copyrighted  images  or  other  materiai,  permission  from  the  copyright  hoider  may  be 
necessary  if  you  wish  to  reproduce  this  materiai  separateiy. 


GAO’S  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accoimtability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accoimtability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 
MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accoimtability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 

Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 

